Automated testing comprises automating a previously manual testing process. Typically, automated testing tools require that a formalized manual testing process currently exists, e.g. a set of detailed test cases and test scripts, including a set of results expected for a given test case and test environment which provides an initial, known test bed that allows the test cases to be repeated from a known starting place. Test cases are most often derived from a set of system requirements to be verified as being satisfied by the system under test.
Commercially developed software testing tools have design constraints, e.g. they are specifically designed to be Internet web testers, performance testers, regression testers, script generators, test data generators, and the like. These tools have limited customization capabilities. Moreover, these tools are often static with respect to tailorability to a given test bed.
However, application systems and environments differ, e.g. in platform, application requirements, and user tastes and desires. Testing methodologies and procedures also vary greatly and often require adaptability to meet the user's quality needs and certification goals.
A need exists for a standardizable format and/or process that allows integration of software application testing tools and the software application testing process with other applications such as office applications, e.g. word processors, spreadsheets, and the like. Test results need to be stored in a manner that supports integration with office tools such as word processors. Such integration would further allow sharing and using test systems and their data by a group of people. Sharing is often needed when a group collaborates in a testing effort.
There is also a need for flexible reporting, e.g. user tailorable, ad hoc result reports.
System test tools also need to be able to gather and/or generate test metrics and make these metrics available for post testing analysis, e.g. metrics comprising comparisons of test data results between two iterations of project versions, integration into the test system of defects found later by users or technical support, reliability calculations, and the like.
Accordingly, a test system which provides for testing to a desired test level and thoroughness but which is adaptable to the system under test is desirable. As opposed to current test systems which most often require the system under test to be architected or otherwise modified to work with a test tool, a test tool should itself be able to be modified and adapted to work with the system under test. Such adaptation should provide for management and customization of the test system, including management and customization of test procedures, resulting test data, and presentation of the resulting test data. Ideally, the user interface that allows access to the test system's functions should be customizable as well to adapt to the system under test.